Terminal device, calibration method and calibration program

ABSTRACT

A terminal device such as a smartphone displays a first image on a display unit, and outputs a second image including the first image as a part to a display device such as a navigation device. Also, the terminal device obtains relative position information indicating a relative positional relation of the first image with respect to the second image from an external server. When a user makes a touch input to the second image displayed on the display device, the input position is transmitted from the display device to the terminal device. The terminal device obtains the conversion necessity information indicating whether or not the input position on the second image needs to be converted based on the relative position information, and converts the input position on the second image to the input position on the first image based on the relative position information when the conversion is necessary.

TECHNICAL FIELD

The present invention relates to calibration of a touch input to aterminal device and a display device connected to each other.

BACKGROUND TECHNIQUE

In recent years, an on-vehicle device such as a car navigation deviceand a terminal device such as a smartphone are connected and used.

For example, Patent Reference 1 discloses a system in which a portableterminal and an on-vehicle device are cooperated and an applicationexecuted on the terminal device is operated from the on-vehicle device.

Non-Patent Reference 1 proposes a technique of transmitting a displayimage of a smartphone to an on-vehicle device to be displayed on adisplay unit of the on-vehicle device and operating an application onthe smartphone by a touch input to the display unit of the on-vehicledevice, by installing a dedicated module to the application installed inthe smartphone.

When the display image of the smartphone is transmitted to the displaydevice and displayed like Non-Patent Reference 1, the number of pixelsand the aspect ratio of the display images are not necessarily the samebetween the smartphone and the display device. Therefore, according tothe model of the smartphone, a black frame area is added at the top,bottom, left and right part of the outputted image in order to maintainthe aspect ratio of the outputted image. Namely, the image displayed onthe touch panel of the display device is the image created by adding theblack frame area to the top, bottom, left and right part of the displayimage of the smartphone.

In this case, since the black frame area is included in the displayimage, the display device cannot discriminate whether or not the blackframe area is being displayed. Therefore, when a part of the black framearea is touched on the touch panel of the display device, the smartphonemay determine that the area within the display image is touched,although the touched area includes no button and the like. In addition,since the area of the image displayed on the touch panel of the displaydevice does not coincide with the actual touch area, the coordinates ofthe point touched by a user may shift even inside of the black framearea.

In this view, the Patent Reference 2 discloses the method of preparingthe calibration information for each combination of the smartphone andthe display device and converting the input coordinates inputted by theuser on the display device to the input coordinates of the smartphonebased on the calibration information.

PRIOR ART REFERENCE Patent Reference

-   Patent Reference 1: Japanese Patent Application Laid-open under No.    2010-130670-   Patent Reference 2: International Publication WO2013/150637

Non-Patent Reference

-   Non-Patent Reference 1: AppRadio Pioneer Electronics USA    http://www.pioneerelectronics.com/PUSA/Car/AppRadio

SUMMARY OF INVENTION Problem to be Solved by the Invention

However, there exist some display devices loaded on a vehicle, that hasa function to individually cope with the black frame being displayed asdescribed above. For the display device having such a function, thecoordinate conversion processing like Patent Reference 2 is notnecessary. Also, if the smartphone executes the processing like PatentReference 2 after the display device individually executes theprocessing for the black frame, there may occur such a problem that thecoordinates that the smartphone recognize are shifted.

The above is one example of a problem to be solved by the presentinvention. It is an object of the present invention to enable easycorrection of a shift of an input position caused by the shift of thedisplay images of the terminal device and the display device, accordingto the function of the display device.

Means for Solving the Problem

The invention described in claims is a terminal device comprising: adisplay unit configured to display a first image; an image output unitconfigured to output a second image including the first image as a partto a display device; an input position obtaining unit configured toobtain, from the display device, an input position on the second imageindicating a position of a touch input that a user made to the secondimage displayed on the display device; a relative position informationobtaining unit configured to obtain, from an external server, a relativeposition information indicating a relative positional relation of thefirst image with respect to the second image; a conversion necessityinformation obtaining unit configured to obtain, from the displaydevice, a conversion necessity information indicating whether or not theinput position on the second image needs to be converted based on therelative position information; and a converter configured to convert theinput position on the second image to the input position on the firstimage based on the relative position information when the conversionnecessity information obtaining unit obtains the conversion necessityinformation indicating that the conversion is necessary.

The invention described in claims is a calibration method executed by aterminal device including a display unit, comprising: a display processwhich displays a first image; an image output process which outputs asecond image including the first image as apart to a display device; aninput position obtaining process which obtains, from the display device,an input position on the second image indicating a position of a touchinput that a user made to the second image displayed on the displaydevice; a relative position information obtaining process which obtains,from an external server, a relative position information indicating arelative positional relation of the first image with respect to thesecond image; a conversion necessity information obtaining process whichobtains, from the display device, a conversion necessity informationindicating whether or not the input position on the second image needsto be converted based on the relative position information; and aconversion process which converts the input position on the second imageto the input position on the first image based on the relative positioninformation when the conversion necessity information obtaining unitobtains the conversion necessity information indicating that theconversion is necessary.

The invention described in claims is a calibration program executed by aterminal device including a display unit and a computer, the programmaking the computer execute: a display process which displays a firstimage; an image output process which outputs a second image includingthe first image as a part to a display device; an input positionobtaining process which obtains, from the display device, an inputposition on the second image indicating a position of a touch input thata user made to the second image displayed on the display device; arelative position information obtaining process which obtains, from anexternal server, a relative position information indicating a relativepositional relation of the first image with respect to the second image;a conversion necessity information obtaining process which obtains, fromthe display device, a conversion necessity information indicatingwhether or not the input position on the second image needs to beconverted based on the relative position information; and a conversionprocess which converts the input position on the second image to theinput position on the first image based on the relative positioninformation when the conversion necessity information obtaining unitobtains the conversion necessity information indicating that theconversion is necessary.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system according to an embodiment of the presentinvention.

FIG. 2 is a block diagram illustrating a functional configuration of thesystem when a display device without an image size adjusting function isused.

FIG. 3 illustrates an example of an image to which a black frame isadded to adjust an aspect ratio.

FIG. 4 illustrates an example of an input position conversion method.

FIG. 5 is a block diagram illustrating a functional configuration of thesystem when a display device with an image size adjusting function isused.

FIG. 6 is a flowchart of input position conversion processing.

FIG. 7 illustrates an example of a calibration image.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

According to one aspect of the present invention, there is provided aterminal device comprising: a display unit configured to display a firstimage; an image output unit configured to output a second imageincluding the first image as a part to a display device; an inputposition obtaining unit configured to obtain, from the display device,an input position on the second image indicating a position of a touchinput that a user made to the second image displayed on the displaydevice; a relative position information obtaining unit configured toobtain, from an external server, a relative position informationindicating a relative positional relation of the first image withrespect to the second image; a conversion necessity informationobtaining unit configured to obtain, from the display device, aconversion necessity information indicating whether or not the inputposition on the second image needs to be converted based on the relativeposition information; and a converter configured to convert the inputposition on the second image to the input position on the first imagebased on the relative position information when the conversion necessityinformation obtaining unit obtains the conversion necessity informationindicating that the conversion is necessary.

The above terminal device such as a smartphone displays a first image ona display unit, and outputs a second image including the first image asa part to a display device such as a navigation device. Also, theterminal device obtains relative position information indicating arelative positional relation of the first image with respect to thesecond image from an external server. When a user makes a touch input tothe second image displayed on the display device, the input position istransmitted from the display device to the terminal device. The terminaldevice obtains the conversion necessity information indicating whetheror not the input position on the second image needs to be convertedbased on the relative position information, and converts the inputposition on the second image to the input position on the first imagebased on the relative position information when the conversion isnecessary. Thereby, when the specifications are different between thedisplay unit of the terminal device and the display unit of the displaydevice and the coordinate conversion is necessary on the terminal deviceside, the input position to the display device can be correctlyconverted to the input position to the terminal device. On the otherhand, when the display device has the coordinate conversion function andthe coordinate conversion on the terminal device side is not necessary,the coordinate conversion is not performed. Therefore, it can beprevented that the input position is shifted by the unnecessarycoordinate conversion.

In one mode of the above terminal device, the relative positioninformation obtaining unit stores the relative position informationobtained from the external server into a storage unit in the terminaldevice, and the converter uses the relative position information storedin the storage unit when the relative position information obtained inthe past is stored in the storage unit. Thereby, it becomes unnecessaryto obtain the relative position information every time from the externalserver, and the communication amount can be reduced.

Another mode of the above terminal device further comprises: acalibration image output unit configured to supply a calibration imageto the display device; a calibration input position obtaining unitconfigured to obtain, from the display device, a calibration inputposition indicating a position of the touch input that the user made tothe calibration image displayed on the display device; a relativeposition information generating unit configured to generate the relativeposition information based on the obtained calibration input position;and a communication unit configured to transmit the generated relativeposition information to the external server.

In this mode, when the relative position information cannot be obtainedfrom the external server, the relative position information can begenerated based on the input by the user to the calibration image. Also,by uploading the generated relative position information to the server,many users can share the relative position information.

According to another aspect of the present invention, there is provideda calibration method executed by a terminal device including a displayunit, comprising: a display process which displays a first image; animage output process which outputs a second image including the firstimage as a part to a display device; an input position obtaining processwhich obtains, from the display device, an input position on the secondimage indicating a position of a touch input that a user made to thesecond image displayed on the display device; a relative positioninformation obtaining process which obtains, from an external server, arelative position information indicating a relative positional relationof the first image with respect to the second image; a conversionnecessity information obtaining process which obtains, from the displaydevice, a conversion necessity information indicating whether or not theinput position on the second image needs to be converted based on therelative position information; and a conversion process which convertsthe input position on the second image to the input position on thefirst image based on the relative position information when theconversion necessity information obtaining unit obtains the conversionnecessity information indicating that the conversion is necessary.Thereby, when the specifications are different between the display unitof the terminal device and the display unit of the display device andthe coordinate conversion is necessary on the terminal device side, theinput position to the display device can be correctly converted to theinput position to the terminal device.

According to still another aspect of the present invention, there isprovided a calibration program executed by a terminal device including adisplay unit and a computer, the program making the computer execute: adisplay process which displays a first image; an image output processwhich outputs a second image including the first image as a part to adisplay device; an input position obtaining process which obtains, fromthe display device, an input position on the second image indicating aposition of a touch input that a user made to the second image displayedon the display device; a relative position information obtaining processwhich obtains, from an external server, a relative position informationindicating a relative positional relation of the first image withrespect to the second image; a conversion necessity informationobtaining process which obtains, from the display device, a conversionnecessity information indicating whether or not the input position onthe second image needs to be converted based on the relative positioninformation; and a conversion process which converts the input positionon the second image to the input position on the first image based onthe relative position information when the conversion necessityinformation obtaining unit obtains the conversion necessity informationindicating that the conversion is necessary. Thereby, when thespecifications are different between the display unit of the terminaldevice and the display unit of the display device and the coordinateconversion is necessary on the terminal device side, the input positionto the display device can be correctly converted to the input positionto the terminal device.

EMBODIMENTS

A preferred embodiment of the present invention will be described belowwith reference to the attached drawings.

[Basic Configuration]

FIG. 1 illustrates a configuration of a system including a terminaldevice according to an embodiment of the present invention. As shown inFIG. 1, the system includes a server 1, a smartphone 10 and a displaydevice 20.

The smartphone 10 and the display device 20 are configured to be able tocommunicate with each other. The smartphone 10 corresponds to theterminal device of the present invention. The display device 20 is adevice loaded on a vehicle, such as a navigation device or an on-vehicleAV equipment.

In this system, the display image of the smartphone 10 is transmitted tothe display device 20 and is displayed on the display device 20. When auser makes an operational input to the display device 20, the displaydevice 20 transmits a signal corresponding to the operational input tothe smartphone 10. Thus, the user can operate the smartphone 10 bymaking the operational input to the display device 20.

The smartphone 10 displays the image on a touch panel (display unit) 14using a liquid crystal display, for example. The smartphone 10 transmitsthe image D1 displayed on the touch panel 14 to the display device 20via HDMI (High Definition Multimedia Interface) (Registered Trademark),for example.

The display device 20 includes a touch panel (display device) 21 using aliquid crystal display, for example. The display device 20 receives theimage D1 from the smartphone 10 and displays it on the touch panel 21.Also, the display device 20 detects the operational input that the usermade to the touch panel 21, and transmits its input position (positioncoordinates) D2 to the smartphone 10. The input position D2 istransmitted to the smartphone 10 by SPP (Serial Port Profile) ofBluetooth (Registered Trademark), for example.

The server 1 includes a database 2. The database 2 stores relativeposition information D3 (hereinafter also referred to as “calibrationinformation D3”). The smartphone 10 accesses the server 1 by a wirelesscommunication function, and is able to download the calibrationinformation D3.

[In Case of Display Device without Image Size Adjusting Function]

FIG. 2 is a block diagram illustrating a functional configuration of thesystem shown in FIG. 1 when a display device 20A without an image sizeadjusting function is used. In the following description, “the displaydevice 20A” indicates the display device without the image sizeadjusting function, “the display device 20B” indicates the displaydevice with the image size adjusting function, and “the display device20” includes both of them.

The image size adjusting function is such a function that a userarbitrarily cuts out a part of an image displayed on a display device,enlarges the image thus cut out to the whole size of the screen anddisplays it, for example. In the example of FIG. 2, since the displaydevice 20A does not have the image size adjusting function, thesmartphone 10 performs the following coordinate conversion to adjust thesize of the image displayed on the smartphone 10 and the size of theimage displayed on the display device 20A.

The smartphone 10 includes an application 11, an image output unit 12, acontrol unit 13, the touch panel 14 and communication units 15 and 16.

The application 11 is executed on the smartphone 10, and is used tooperate the smartphone 10 by the display device 20 as described above.The application 11 is realized by a computer such as a CPU whichexecutes a program prepared in advance. The application 11 functions asan input position obtaining unit, a calibration information obtainingunit, a calibration image output unit, a calibration positioninformation obtaining unit, a conversion necessity information obtainingunit and a calibration information generation unit according to thepresent invention.

The image output unit 12 receives the display image of the smartphone 10from the application 11, and transmits it to the display device 20 asthe image D1.

The control unit 13 controls the smartphone 10 in its entirety, and isactually realized by an OS and a basic application installed in thesmartphone 10. The control unit 13 displays the image D12 generated byvarious applications operating on the smartphone 10 on the touch panel14 and also supplies the image D12 to the application 11.

The communication unit 15 has a communication function to perform thewireless communication with the server 1 and receive the calibrationinformation D3 from the server 1. The communication unit 16 has acommunication function to receive the input position D2 corresponding tothe user's operational input from the display device 20.

Meanwhile, the display device 20A includes the touch panel 21, acommunication unit 22, an input unit 23 and a control unit 25. The touchpanel 21 is used when the user makes a desired operational input. Theinput unit 23 detects the operational input that the user made to thetouch panel 21, and supplies its input position D2 to the communicationunit 22. The communication unit 22 transmits the input position D2 tothe smartphone 10.

The control unit 25 controls the display device 20A in its entirety.Also, the control unit 25 includes a memory 26. The memory 26 storesinformation indicating whether or not the display device itself has theimage size adjusting function. The information indicating whether or notthe display device has the image size adjusting function may be set as aflag (hereinafter referred to as “Adjusting Function Flag”). As anexample, the display device that has the image size adjusting functionstores “Adjusting Function Flag=1” in its memory 26. On the contrary,the display device that does not have the image size adjusting functionstores “Adjusting Function Flag=0” in its memory 26. In the example ofFIG. 2, since the display device 20A does not have the image sizeadjusting function, the memory 26 stores “Adjusting Function Flag=0” inthe memory 26.

When the display device 20A is connected to the smartphone 10, thecontrol unit 25 notifies the smartphone 10 of the necessity of thecoordinate conversion on the smartphone 10 side based on the value ofthe Adjusting Function Flag stored in the memory 26. In the example ofFIG. 2, since the display device 20A does not have the image sizeadjusting function and hence the smartphone 10 need to perform thecoordinate conversion so as to absorb the difference between the imagesize of the smartphone 10 and the image size of the display device 20A,the control unit 25 notifies the smartphone 10 of the necessity of thecoordinate conversion on the smartphone 10 side (“Coordinate Conversion:Necessary”).

Next, the processing executed by the application 11 will be described.The application 11 receives the image D12 generated by variousapplications executed on the smartphone 10 from the control unit 13, andsupplies it to the image output unit 12. The image output unit 12transmits the image D12 to the display device 20A as the image D1. Thedisplay device 20A displays the received image D1 on the touch panel 21.Thus, the display image of the smartphone 10 is displayed on the touchpanel 21 of the display device 20A.

The user watches the display image on the display device 20A and makesthe operational input to the touch panel 21 of the display device 20A.Specifically, the user makes the operational input by touching thebutton in the image displayed on the touch panel 21. This operationalinput is detected by the input unit 23, and the input position D2corresponding to the operational input is transmitted to the smartphone10 via the communication unit 22.

In the smartphone 10, the communication unit 16 receives the inputposition D2 and supplies it to the application 11. The application 11obtains the input position of the user's operational input to thesmartphone 10, and supplies the input position to the control unit 13.The control unit 13 executes the processing corresponding to the user'soperational input based on the display image D12 and the input positionreceived from the application 11. Thus, the smartphone 10 can beoperated from the display device 20.

(Conversion Method of Input Position)

When the image of the smartphone 10 is transmitted to and displayed onthe display device 20, it is not necessarily ensured that thespecifications, specifically the number of pixels and the aspect ratio,of the touch panel 14 of the smartphone 10 and the touch panel 21 of thedisplay device 20 coincide with each other. Therefore, there is a casethat the smartphone 10 adds a black frame at the top, bottom, left andright part of the image in order to adjust the number of pixels and theaspect ratio. For example, the aspect ratio of the transmitted image isfixed to 16:9 or 4:3 according to the HDMI standard mentioned above,whereas the aspect ratio of the touch panel 14 of the smartphone 10 isdifferent according to the model of the smartphone 10. Therefore, whenoutputting the image according to HDMI, the smartphone 10 may add theblack frame different according to the model by the image output unit 12thereby to adjust the aspect ratio.

FIG. 3 illustrates an example of the image to which the black frame isadded in order to adjust the aspect ratio. The image IM is displayed onthe touch panel 14 of the smartphone 10. The image IM includes a list ofplural songs belonging to a certain album and the “BACK” button toreturn to the previous screen. When this image IM is transmitted to anddisplayed on the display device 20A, the smartphone 10 creates the imageIMx by adjusting the aspect ratio to be in conformity with the HDMIstandard, and transmits the image IMx to the display device 20A as theimage D1. Thus, the image IMx displayed on the touch panel 21 of thedisplay device 20A is the image created by adding the black frame areaBK to the outer circumference of the image IM displayed on thesmartphone 10.

In this case, since the black frame area BK is also displayed on thetouch panel 21, the user can make the touch input to the black framearea BK. However, since the black frame area Bk does not exist in theimage IM of the smartphone 10 and is not the operational input area, thetouch input to the black area Bk should be treated as an inappropriateinput. In addition, since the image IM displayed on the smartphone 10and the image IMx displayed on the display device 20A have differentscales, it is necessary to convert the position coordinates of the touchinput to the area inside the black frame area BK to the positioncoordinates on the image IM of the smartphone 10.

Here, when the display device 20 has the image size adjusting function,the input position D2 transmitted from the display device 20 is alreadyadapted to the image size of the smartphone 10, and it is not necessaryfor the smartphone 10 to perform the conversion processing of theposition coordinates. On the contrary, when the display device 20A doesnot have the image size adjusting function as in the example of FIG. 2,the input position D2 transmitted from the display device 20A is adaptedto the image size of the display device 20A but is not adjusted to theimage size of the smartphone 10. Therefore, in this embodiment, when thedisplay device 20A does not have the image size adjusting function andthe smartphone 10 receives the coordinate conversion necessityinformation of “Coordinate Conversion: Necessary” from the displaydevice 20A, the application 11 of the smartphone 10 functions as theconverter 11 x to perform this conversion processing.

Specifically, the converter 11 x receives the input position D2 on thetouch panel 21 (i.e., the input position on the image IMx) from thedisplay device 20A, and converts it to the input position on the touchpanel 14 of the smartphone 10 (i.e., the input position on the imageIM). At this time, the converter 11 x mainly executes the following twoprocessing.

First, when the touch input to the black frame area BK is a simple touch(i.e., a type of touch that contacts substantially one point and is thenreleased without the movement of the touched position), the converter 11x simply neglects the touch input. In contrast, when the touch input tothe black frame area Bk is the input involving the movement of thetouched position, such as a so-called “drag”, the converter 11 xconverts the input position of the touch input to the black frame areaBK to the position nearest to the input position in the area inside theblack frame area BK (hereinafter referred to as “regular area”). Byeither of the above methods, the converter 11 x handles the touch inputto the black frame area BK.

Second, the converter 11 x converts the input position of the touchinput to the regular area to the corresponding position on the touchpanel 14 of the smartphone 10. By this, even if the number of pixels ofthe touch panel 14 of the smartphone 10 is different from the number ofpixels of the touch panel 21 of the display device 20A, the inputposition to the touch panel 21 of the display device 20A can becorrectly converted to the input position to the touch panel 14 of thesmartphone 10.

FIG. 4 illustrates an example of the above-described conversion method.First, the horizontal direction will be described. It is assumed thatthe dot number of the touch panel 21 of the display device 20 in thehorizontal direction is “W”, and each of the left and right widths ofthe black frame area BK displayed on the touch panel 21 are “Wa” and“Wb”. Also, it is assumed that the dot number of the touch panel 14 ofthe smartphone 10 in the horizontal direction is “Ws”.

In this case, the input position “X” on the touch panel 21 of thedisplay device 20A is converted to the input position “xs” on the touchpanel 14 of the smartphone 10 by the following equation:

_i xs=0:when X<Wa,

xs=Ws·(X−Wa)/{W−(Wa+Wb)}:when Wa<X<Wb,

xs=Ws:when X>(W−Wb)

The same may apply to the vertical direction. It is assumed that the dotnumber of the touch panel 21 of the display device 20 in the verticaldirection is “H”, each of the top and bottom widths of the black framearea BK displayed on the touch panel 21 are “Hb” and “Ha”, and the dotnumber of the touch panel 14 of the smartphone 10 in the verticaldirection is “Hs”.

In this case, the input position “Y” on the touch panel 21 of thedisplay device 20 is converted to the input position “ys” on the touchpanel 14 of the smartphone 10 by the following equation:

ys=0:when Y<Ha,

ys=Hs·(X−Ha)/{H−(Ha+Hb)}:when Ha<Y<Hb,

ys=Hs:when Y>(H−Hb)

It is noted that, when the touch input to the black frame area BK is thesimple touch, the converter 11 x may neglect the touch input asmentioned above.

According to the above-described conversion method, even when the imageIMx including the black frame area BK is displayed on the touch panel 21of the display device 20A, the input position to the touch panel 21 canbe correctly converted to the input position to the touch panel 14 ofthe smartphone 10.

In order to perform the above-described conversion, the dot numbers (thedot number in the vertical direction: Hs, the dot number in thehorizontal direction: Ws) of the touch panel 14 of the smartphone 10,the dot numbers (the dot number in the vertical direction: H, the dotnumber in the horizontal direction: W) of the touch panel 21 of thedisplay device 20A, and the widths of the top, bottom, left and rightpart of the black frame area BK (Wa, Wb, Ha, Hb) are needed. Theapplication 11 can obtain the dot numbers of the touch panel 14 of thesmartphone 10 from the control unit 13 of the smartphone 10, and canobtain the dot numbers of the touch panel 21 of the display device 20from the display device 20A. However, the widths of the top, bottom,left and right part of the black frame area BK are not normally owned bythe smartphone 10 and the display device 20A.

Therefore, the application 11 obtains the widths of the top, bottom,left and right part of the black frame area BK from the server 1 as thecalibration information D3. Specifically, the widths of the top, bottom,left and right part of the black frame area BK is determined based onthe specification of the touch panel 14 of the smartphone 10 and thespecification of the touch panel 21 of the display device 20A.Therefore, the server 1 retains the widths of the top, bottom, left andright part of the black frame area BK in correspondence with thecombination of the smartphone 10 and the display device 20A as thecalibration information D3 in advance. For example, the server 1 storesthe calibration information D3 indicating the widths of the top, bottom,left and right part of the black frame area BK in correspondence withthe combination of the ID indicating the model name of the smartphone 10and the ID indicating the model name of the display device 20A. When thesmartphone 10 is connected to the display device 20A, the smartphone 10obtains the ID of the display device 20, transmits the ID of thesmartphone 10 and the ID of the display device 20 to the server 1, andobtains the calibration information D3 corresponding to the combinationof those IDs. Then, the smartphone 10 uses the obtained calibrationinformation D3 to convert the input position to the touch panel 21 ofthe display device 20A to the input position to the touch panel 14 ofthe smartphone 10.

[In Case of Display Device with Image Size Adjusting Function]

FIG. 5 is a block diagram illustrating a functional configuration of thesystem shown in FIG. 1 when the display device 20B with the image sizeadjusting function is used. The image size adjusting function is such afunction that a user arbitrarily cuts out a part of an image displayedon a display device, enlarges the image thus cut out to the whole sizeof the screen and displays it, for example. As is understood by thecomparison with FIG. 2, the display device 20B in FIG. 5 is differentfrom the display device 20A shown in FIG. 2 in that it includes aconverter 27 to convert the input coordinates outputted by the inputunit 23 and supply them to the communication unit 22 as the inputcoordinates D2. Other than this, the display device 20B is the same asthe display device 20A shown in FIG. 2. Also, in FIG. 5, the server 1and the smartphone 10 are the same as those in FIG. 2.

As described above, when the smartphone 10 transmits the image to thedisplay device 20B, it may add the black frame to the top, bottom, leftand right area of the image so as to adjust the number of pixels and theaspect ratio. In that case, since the black frame added by thesmartphone 10 is a part of the image data, it is displayed on thedisplay unit 21 of the display device 20B as it is. Therefore, the useruses the image size adjusting function of the display device 20B toenlarge the image D1 transmitted from the smartphone 10 such that theblack frame is not displayed (by cutting out the image area other thanthe black frame). Thus, the black frame added by the smartphone 10 isnot shown on the display unit 21 of the display device 20B.

In addition, the display device 20B recognizes how much the user hasenlarged the display image by the image size adjusting function, and theconverter 27 converts the user's input position on the display imageafter the enlargement to the coordinates of the input position on thedisplay image before the enlargement (i.e., the input positioncorresponding to the image size of the smartphone 10). Thus, when theuser has enlarged the display image to an arbitrary size, the inputposition (input coordinates) of the user's touch input can be correctlyconverted to the input position on the image size of the smartphone 10.The communication unit 22 transmits, to the smartphone 10, the inputposition D2 after being converted to fit the image size of thesmartphone 10 by the converter 27. In this way, the coordinateconversion on the smartphone 10 side becomes unnecessary.

Since the display device 20B shown in FIG. 5 has the image sizeadjusting function, the memory 26 of the control unit 25 stores“Adjusting Function Flag=1”. Therefore, the control unit 25 notifies thesmartphone 10 of the coordinate conversion necessity informationindicating that the coordinate conversion on the smartphone 10 side isunnecessary (“Coordinate Conversion: Unnecessary”). Thus, the smartphone10 does not perform the coordinate conversion, and it becomes possibleto prevent the smartphone 10 from performing unnecessary coordinateconversion even though the display device 20B transmits the inputposition D2 correctly adapted to the image size of the smartphone 10.

Further, the display device 20B may obtain the calibration informationD3 from the smartphone 10 via the communication unit 22, and theconverter 27 may perform the conversion processing described above byusing the obtained calibration information D3, for example. With thisconfiguration, even if the user cuts out and enlarges the image areapartly including the black frame by the image size adjusting function,the appropriate coordinate conversion between the display device 20B andthe smartphone 10 can be performed.

[Input Position Conversion Processing]

FIG. 6 is a flowchart of input position conversion processing. Thisprocessing is executed by the application 11 in the smartphone 10. It isassumed that the smartphone 10 and the display device 20 can communicatewith each other. This processing is executed when the application 11 isactivated on the smartphone 10.

First, when the application 11 is activated, the application 11 requeststhe display device 20 to transmit the coordinate conversion necessityinformation, and receives the coordinate conversion necessityinformation from the control unit 25 of the display device 20 (step S9).As described above, the coordinate conversion necessity information isdetermined based on whether or not the display device 20 has the imagesize adjusting function. When the coordinate conversion necessityinformation received from the display device 20 is “CoordinateConversion: Unnecessary” (step S10: No), the coordinate conversion onthe smartphone 10 side is unnecessary, and the application 11 ends theprocessing.

On the other hand, when the coordinate conversion necessity informationreceived from the display device 20 is “Coordinate Conversion:Necessary” (step S10: Yes), the application 11 determines whether it isthe first time activation or not (step S11). This determination is madebased on processing history information stored in the application 11,for example.

When it is the first time activation (step S11; Yes), normally theapplication 11 does not have the calibration information yet. Therefore,the application 11 specifies the ID of the smartphone 10 and the ID ofthe display device 20 to request the calibration informationcorresponding to the combination of those IDs, and obtains thecorresponding calibration information to store it in the storage unit ofthe smartphone 10 (step S12). Then, the processing goes to step S14.

On the other hand, when it is not the first time activation (step S11;No), the application 11 determines whether or not the calibrationinformation exists in the storage unit of the smartphone 10 (step S13).The calibration information in this case means the calibrationinformation corresponding to the combination of the smartphone 10 andthe display device 20 currently used. When the calibration informationdoes not exist (step S13; No), the application 11 obtains thecalibration information from the server 1 (step S12). On the other hand,when the calibration information exists (step S13; Yes), the processinggoes to step S14.

In step S14, the application 11 determines whether or not thecalibration information is successfully obtained. Namely, theapplication 11 determines whether or not the calibration information isprepared by the processing of step S12 or S13. When the application 11successfully obtains the calibration information (step S14; Yes), theprocessing goes to step S15.

On the other hand, when the application does not successfully obtain thecalibration information (step S14; No), the application 11 executes theprocessing of generating the calibration information by itself byutilizing the display device 20. Specifically, the application 11transmits the calibration image to the display device 20 to display thecalibration image on the touch panel 21 of the display device 20 (stepS16). As the reason why the calibration information is not successfullyobtained, there may be a case where the server 1 does not have thecalibration information corresponding to the combination of thesmartphone 10 and the display device 20, or a case where thecommunication with the server 1 cannot be normally executed.

FIG. 7 shows an example of the calibration image. In the example of FIG.7, the application 11 transmits the image, expressing the regular areaby the light and shade in order to distinguish it from the black framearea BK, to the display device 20 as the calibration image. In thiscase, the calibration image includes the black frame area BK based onthe difference of the number of pixels and/or the aspect ratio betweenthe touch panel 14 of the smartphone 10 and the touch panel 21 of thedisplay device 20. Further, the application 11 indicates the inner fourcorners of the black frame area BK by the marks C1 to C4 to make theuser touch those inner four corners of the black frame area BK. When theuser touches the positions of the marks C1 to C4, the display device 20acquires the coordinates of each position, and transmits them to thesmartphone 10.

The application 11 receives the position coordinates corresponding tothe inner four corners of the black frame area BK, generates thecalibration information by calculating the widths of the top, bottom,left and right part of the black frame area BK, and then stores it inthe storage unit of the smartphone 10 (step S17). Further, theapplication 11 uploads the calibration information thus generated to theserver 1 together with the ID of the smartphone 10 and the ID of thedisplay device 20 (step S18). By this, the calibration informationcorresponding to the combination of the smartphone and the displaydevice is prepared in the server 1, and it can be used by other usersafter that time. Then, the processing goes to step S15.

In step S15, the application 11 converts the input position by using theprepared calibration information (step S15). Namely, the application 11obtains the input position to the touch panel 21 of the display device20 from the display device 20, and converts it to the input position tothe touch panel 14 of the smartphone 10. Then, the application 11supplies the input position D13 obtained by the conversion to thecontrol unit 13 of the smartphone 10, and ends the conversionprocessing. It is noted that the control unit 13 of the smartphone 10processes the operational input by the user based on the input positionD13.

In this way, even when the specification such as the number of pixelsand/or the aspect ratio is different between the touch panel 14 of thesmartphone 10 and the touch panel 21 of the display device 20 andtherefore the black frame area BK is displayed in the display image ofthe display device 20, the input position to the touch panel 21 of thedisplay device 20 can be correctly converted to the input position tothe touch panel 14 of the smartphone 10.

When the calibration information used to convert the input position isstored in the smartphone 10, the application 11 uses it, and thereforethe application 11 does not have to download the calibration informationfrom the server 1. In a case where the application 11 cannot obtain thecalibration information from the server 1, the application 11 generatesthe calibration information based on the input by the user by using thecalibration image as shown in FIG. 7. Therefore, even in case of using asmartphone or a display device of new model whose calibrationinformation does not exist yet in the server 1, the conversion of theinput position is possible. Generating the calibration information byusing the calibration image as described above is performed not only ina case where the application 11 cannot obtain the calibrationinformation. For example, the processing of generating the calibrationinformation may be registered in a menu of the smartphone 10 and may beperformed when the user desires.

Since the application 11 uploads the calibration information thusgenerated to the server 1, other users can use the calibrationinformation after that. If the server 1 receives the calibrationinformation from plural smartphones, it may retain the calibrationinformation of high accuracy by analyzing the accuracy of them and/oraveraging them.

Modified Example

In the embodiment described above, the control unit 25 transmits thecoordinate conversion necessity information of “Coordinate Conversion:Unnecessary” to the smartphone 10 when the display device 20 has theimage size adjusting function, and transmits the coordinate conversionnecessity information of “Coordinate Conversion: Necessary” to thesmartphone 10 when the display device 20 does not have the image sizeadjusting function.

However, even if the display device 20 has the image size adjustingfunction, when the user is displaying the image D1 transmitted from thesmartphone 10 on the display unit 21 as it is, without actually usingthe image size adjusting function (e.g., the image size adjustingfunction is being set to “OFF” state), the control unit 25 may transmitthe coordinate conversion necessity information of “CoordinateConversion: Necessary” to the smartphone 10 to perform the coordinateconversion on the smartphone 10 side.

Also, when the image size adjusting function on the display device 20side includes not only the enlargement of the entire image but theenlargement only in vertical or horizontal direction, the control unit25 may notify the smartphone 10 of the necessity of the coordinateconversion of the vertical or horizontal direction as the coordinateconversion necessity information. For example, when the user isperforming the size adjustment only in the vertical direction and is notperforming the size adjustment in the horizontal direction by the imagesize adjusting function of the display device 20, the control unit 25may transmit the coordinate conversion necessity information of“Vertical Direction Coordinate Conversion: Unnecessary, HorizontalDirection Coordinate Conversion: Necessary” to the smartphone 10.

INDUSTRIAL APPLICABILITY

This invention can be used in a case where a terminal device such as asmartphone is connected to a display device, not only an on-vehicledisplay device, having an operation unit of a touch input.

DESCRIPTION OF REFERENCE NUMBERS

-   -   1 Server    -   10 Smartphone    -   11 Application    -   11 x, 24 Converter    -   12 Image Output Unit    -   13 Control Unit    -   14, 21 Touch Panel    -   20 Display Device

1.-7. (canceled)
 8. A system comprising a terminal device and a displaydevice communicable with each other, wherein the terminal devicecomprises: a display unit configured to display a first image; an imageoutput unit configured to output a second image including the firstimage to the display device; and an obtaining unit configured to obtain,from the display device, information related to specification of a touchpanel of the display device, the image output unit determiningspecification of the second image based on the information related tothe specification of the touch panel, wherein the display devicecomprises: an adjustment unit configured to adjust an image size of thesecond image; the touch panel configured to display the second imageadjusted by the adjustment unit; an input unit configured to detect aninput position of an operational input to the touch panel by a user; aconversion unit configured to convert the input position to a convertedinput position which is an input position on the second image beforeadjustment; and a communication unit configured to transmit theconverted input position to the terminal device, wherein the terminaldevice further comprises a control unit configured to execute processingcorresponding to the operational input in accordance with the convertedinput position.
 9. The system according to claim 8, wherein the imageoutput unit determines a number of pixels or an aspect ratio of thesecond image based on the information related to the specification ofthe touch panel obtained from the display device.
 10. The systemaccording to claim 8, wherein the image output unit generates the secondimage by adding a black frame to the first image.
 11. The systemaccording to claim 8, wherein the image output unit generates the secondimage by adding a black frame to the first image based on theinformation related to the specification of the touch panel obtainedfrom the display device.
 12. The system according to claim 8, whereinthe adjustment unit adjusts the image size by enlarging a part of thesecond image.
 13. The system according to claim 8, wherein theadjustment unit adjusts the image size by enlarging a part of the secondimage such that a black frame of the second image is not displayed. 14.The system according to claim 8, wherein the adjustment unit switches ONand OFF a function of adjusting the image size.
 15. The system accordingto claim 8, wherein the adjustment unit adjusts the image size byenlarging a part of the second image, and wherein the adjustment unitswitches ON and OFF a function of adjusting the image size.
 16. Thesystem according to claim 8, wherein the second image is transmittedfrom the terminal device to the display device via an HDMI, and whereinthe converted input position is transmitted from the display device tothe terminal device via Bluetooth.
 17. The system according to claim 8,wherein the image output unit generates the second image by adding ablack frame to the first image based on the information related to thespecification of the touch panel obtained from the display device,wherein the adjustment unit adjusts the image size by enlarging a partof the second image, and wherein the adjustment unit switches ON and OFFa function of adjusting the image size.
 18. The system according toclaim 17, wherein the second image is transmitted from the terminaldevice to the display device via an HDMI, and wherein the convertedinput position is transmitted from the display device to the terminaldevice via Bluetooth.
 19. A terminal device communicable with a displaydevice, comprising: a display unit configured to display a first image;an image output unit configured to output a second image including thefirst image to the display device; an obtaining unit configured toobtain, from the display device, information related to specification ofa touch panel of the display device, a control unit configured toreceive a position signal corresponding to an operational input to thetouch panel by a user and executes processing corresponding to theoperational input based on the position signal, wherein the image outputunit determines specification of the second image based on theinformation related to specification of the touch panel.
 20. Theterminal device according to claim 19, wherein the image output unitdetermines a number of pixels or an aspect ratio of the second imagebased on the information related to the specification of the touch panelobtained from the display device.
 21. The terminal device according toclaim 19, wherein the image output unit generates the second image byadding a black frame to the first image.
 22. The terminal deviceaccording to claim 19, wherein the image output unit generates thesecond image by adding a black frame to the first image based on theinformation related to the specification of the touch panel obtainedfrom the display device.
 23. The terminal device according to claim 19,wherein the second image is transmitted from the terminal device to thedisplay device via an HDMI, and wherein an aspect ratio of the secondimage is selectable from at least 16:9 and 4:3.
 24. A display devicecommunicable with a terminal device, comprising: an adjustment unitconfigured to adjust an image size of a second image; a touch panelconfigured to display the second image adjusted by the adjustment unit;an input unit configured to detect input position of an operationalinput to the touch panel by a user; a conversion unit configured toconvert the input position to a converted input position which is aninput position on the second image before adjustment; and acommunication unit configured to transmit the converted input positionto the terminal device.
 25. The display device according to claim 24,wherein the adjustment unit adjusts the image size by enlarging a partof the second image.
 26. The display device according to claim 24,wherein the adjustment unit switches ON and OFF a function of adjustingthe image size.
 27. The display device according to claim 24, whereinthe adjustment unit adjusts the image size by enlarging a part of thesecond image, and wherein the adjustment unit switches ON and OFF afunction of adjusting the image size.